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Abstract 

This  research  was  performed  for  the  dual  purpose  of  defining 
effectiveness  as  it  applies  to  the  field  of  records  management  (RM)  and 
studying  the  impact  of  Document  Librarian  (DL)  on  the  productivity  of  Air 
Force  records  managers.  The  main  thrust  of  the  study  was  to  define 
effectiveness  for  RM  because  no  industry  standard  had  previously  been 
developed.  The  definition  was  generated  through  the  use  of  a  Delphi  process 
with  a  group  of  RM  experts  serving  as  the  panel.  Four  rounds  of  questioning 
were  administered  during  the  Delphi  before  an  acceptable  consensus  was 
reached. 

The  Delphi  developed  an  effectiveness  definition  by  highlighting  major 
functional  areas  which  a  RM  system  must  be  able  to  successfully  perform. 
The  functions  include  storing  to  eliminate  loss,  screening  records  for 
compliance  with  Freedom  of  Information  Act  and  Privacy  Act  requirements, 
retrieving  documents  by  a  predetermined  field,  grouping  records  for  archival 
storage  in  accordance  with  regulations,  and  indexing  files  by  predetermined 
unique  characteristics. 

DL  is  a  RM  software  package  developed  jointly  by  Air  Force  Material 
Command  (AFMC)  and  Wang  Laboratories.  The  software  went  through  an 
extensive  development  process,  with  the  AFMC  personnel  maintaining  high 
expectations  for  its  performance.  The  impact  of  DL  on  the  productivity  of  RM 
was  tested  through  the  use  of  a  customer  satisfaction  survey.  Survey 
questions  focused  on  areas  such  as  satisfaction  with  reliability,  effect  on  job 
performance,  ease  of  use,  utility,  and  overall  appraisal. 
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The  customer  satisfaction  survey  of  DL  revealed  that  the  software  has 
not  broken  any  records  from  a  user  satisfaction  point  of  view.  Respondents 
indicated  that  they  had  not  significantly  improved  their  RM  productivity 
with  DL’s  capabilities.  They  also  indicated  the  software  is  not  significantly 
easier  to  use  or  learn  or  even  more  user-friendly  than  other  software,  leaving 
the  product's  testers  unsure  as  to  whether  DL's  usefulness  warrants  its 
purchase  and  wider  distribution  throughout  the  Air  Force. 

The  research  team  made  the  following  recommendations.  DL  should  be 
used  for  a  longer  period  of  time  and  by  a  larger  number  of  users  before  its 
performance  is  evaluated.  The  effectiveness  functions  highlighted  by  the 
Delphi  should  be  included  in  any  new  customer  satisfaction  survey  for  DL  as 
a  means  to  verify  that  the  software  adequately  addresses  the  needs  of  the  RM 
community.  Finally,  the  entire  Air  Force  process  of  RM  should  be  reviewed 
for  possible  improvements  before  integrating  RM  software.  This  would  avoid 
the  automation  of  an  already  ineffective  process. 
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AN  ANALYSIS  OF  THE  EFFECTIVENESS  OF  RECORDS  MANAGEMENT 


I.  INTRODUCTION 

The  procurement  and  implementation  of  new  computer-based 
information  systems  is  a  frequent  occurrence  in  the  Air  Force  as  the  service 
strives  to  improve  productivity  in  its  administrative  tasks.  The  Air  Force 
routinely  brings  new  systems  on  line  in  an  effort  to  improve  control  of  its 
records  management  (RM)  process  as  well  as  its  immense  accumulation  of 
paperwork.  Although  managers  use  many  scarce  resources  to  achieve  this 
improvement,  actual  changes  in  productivity  are  seldom  measured,  since  no 
standard  for  measurement  exists. 

The  private  sector  is  used  as  the  primary  source  of  data  for  the 
problem  description  in  this  study  because  of  the  focus  of  existing  research. 
Although  this  thesis  is  sponsored  by  an  Air  Force  organization,  the  research 
team  has  been  unable  to  find  any  Air  Force  data  regarding  the  quantity  of 
records  produced  and  stored,  or  of  RM  productivity. 

As  a  matter  of  course  in  business,  the  United  States  is  inundated  with 
huge  quantities  of  paper.  Every  year  the  private  sector  alone  produces  over 
70  billion  documents,  adding  to  the  nearly  600  billion  documents  already  in 
storage  ("Records  Management  Program...,"  1991:44).  The  effect  of  this 
disposition  is  to  force  management  to  effectively  control  the  maintenance, 
storage,  and  disposal  of  documents  (Jones,  1993:10). 

Improper  handling  of  documents  contributes  greatly  to  lost  profits  and 
costly  litigation,  making  RM  a  critical  function  in  business  (Jones,  1993:10). 
The  need  for  an  extensive  program  of  RM  is  illustrated  by  the  frequent 
occurrence  of  the  following  problems:  "the  severity  of  revenue  loss  due  to  poor 
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records  programs;  prolonged  time  for  retrieval  of  information  from  existing 
records;  increasing  reference  to  active  and  inactive  records;  the  volume  of 
records  created  by  the  business;  the  degree  of  inefficiency  and 
disorganization  of  the  present  filing  and  inactive  records  storage  systems" 
(Jones,  1993:17). 

The  business  environment  has  created  a  voracious  appetite  for  techno¬ 
logically  advanced  products  in  the  RM  field  as  computer-based  systems  use  is 
greater  than  ever  (Hodges,  1993:32).  The  force  of  this  demand  is  so  great  that 
information  technology  is  now  the  largest  sector  for  spending  in  corporate 
America  (Schnitt,  1993:14).  However,  these  programs  are  often  implemented 
without  an  actual  definition  of  productivity  and  effectiveness.  In  addition, 
the  problem  is  confounded  by  the  failure  to  clearly  delineate  the  expected 
benefits  of  these  programs.  Evidence  suggests  that  this  investment  in 
technology  hasn't  produced  the  improvements  desired.  Despite  the  enormous 
expenditures  for  information  technology,  more  paper  is  generated  today  than 
ever  before,  yet  productivity  has  grown  only  1.6  percent  in  the  last  decade 
(Penn,  1992:20). 

One  study  found  that  rather  than  having  more  work  done  using  fewer 
resources,  computers  simply  foster  more  computer-related  activities.  The 
study  came  to  three  sweeping  conclusions: 

•  Businesses  use  much  of  the  information  technology  as  strategic 
management  tools  to  increase  market  share  rather  than  adding  value 
to  products  and  services 

•  Executives  may  generate  more  information  rather  than  reduce  staff 

•  Information  technology  is  not  readily  substitutable  for  information 
labor.  Technology  requires  more  information  workers  as  the  price  of 
technology  falls  (Duncan,  1991:41) 
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A  critical  element  for  records  managers  to  remember  is  the  need  to 
know  in  advance  of  employing  a  new  RM  system  exactly  what  kind  of 
benefits  an  organization  needs  in  order  to  justify  purchase  of  the  system 
(Schnitt,  1993:15,  41).  An  organization  should  also  be  able  to  accurately 
measure  the  effectiveness  of  a  new  system  after  its  implementation.  In  a 
recent  survey,  respondents  were  asked  to  outline  the  steps  they  had  taken  to 
measure  the  productivity  of  their  new  systems.  The  results  showed  that  none 
of  the  managers  polled  did  anything  to  evaluate  a  return  on  investment,  yet 
some  of  them  had  spent  as  much  as  20  percent  of  their  annual  budgets  on  the 
new  technology  (Duncan,  1991:42-43).  "In  the  absence  of  appropriate  tools  to 
assess  the  effectiveness  of  information  technology,  management  will  continue 
to  base  decisions  on  intuition  and  gut  feel"  (Duncan,  1991:44-45). 

At  the  headquarters  of  the  Air  Force  Materiel  Command  (AFMC), 
Wright-Patterson  Air  Force  Base,  Ohio,  the  personnel  in  charge  of  developing 
new  software  systems  for  the  Air  Force  are  aware  of  the  risks  and  the 
frequent  disappointments  of  implementing  new  technology.  Experience  has 
shown  the  problems  often  encountered  when  taking  unnecessary  risks  with 
new  systems.  Furthermore,  they  are  mindful  of  the  fact  that  no  appropriate 
measures  exist  for  defining  effectiveness  in  RM.  To  circumvent  potential 
problems  before  deciding  to  distribute  new  electronic  RM  systems  throughout 
the  Air  Force,  the  AFMC  Corporate  Information  (Cl)  developers 
commissioned  this  study  to  address  the  problem. 

Specific  Problem 

The  purposes  of  this  research  are  to  develop  a  definition  of 
effectiveness  for  the  field  of  RM  and  to  evaluate  the  impact  of  a  modern  office 
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automation  system,  specifically  in  the  field  of  electronic  RM.  The  research 
will  establish  and  define  effectiveness  measures  for  RM  and  evaluate  the 
extent  to  which  one  applicable  electronic  technology  product  enhances  or 
detracts  from  that  effectiveness. 

Investigative  Questions 

1)  How  is  effectiveness  defined  for  RM? 

Developing  a  definition  for  RM  effectiveness  is  the  main  purpose  of 
this  research.  The  goal  is  to  provide  managers  with  a  standard  by  which  any 
potential  acquisition  of  RM  hardware  or  software  can  be  measured.  Since 
resources  are  often  wasted  on  unproductive  RM  systems,  a  definition  of  what 
performance  criteria  a  new  system  must  meet  in  order  to  be  effective  at 
managing  records  should  preclude  such  waste. 

2)  How  does  Document  Librarian  impact  the  process  of  RM? 

The  research  team  will  use  a  survey  to  assess  the  customer  satisfaction 
of  a  new  software  product  developed  for  the  Air  Force.  The  research  sponsor 
commissioned  this  study  to  evaluate  the  impact  of  the  new  system  on  organi¬ 
zational  productivity. 

Simultaneous  research  will  be  conducted  on  both  of  the  investigative 
questions.  As  the  Delphi  process  is  being  accomplished  to  address  question  1, 
the  research  team  will  also  administer  the  survey  to  the  test  population  to 
address  question  2. 
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II.  EFFECTIVENESS  FOR  RECORDS  MANAGEMENT 


As  an  investigation  into  effectiveness  for  the  field  of  RM,  this  chapter 
includes  a  review  of  literature  regarding  effectiveness  and  legal  issues  facing 
records  managers.  The  rest  of  the  chapter  contains  the  methodology  for 
answering  the  first  investigative  question  and  the  results  of  carrying  out  that 
methodology. 

Effectiveness  in  Records  Management 

If  this  study  is  to  define  the  measures  of  effectiveness  for  RM,  then  a 
definition  is  needed  for  exactly  what  is  to  be  measured.  One  source  defined 
work  measurement  in  general  as  "the  time  for  a  qualified  worker  to  carry  out 
a  specified  task  at  a  defined  level  of  performance"  (Boyd,  1992:21).  If  that  is 
true  then  what  task  or  level  of  performance  should  a  records  manager 
measure?  Suggestions  come  from  a  variety  of  sources. 

Weill  and  Olson  conducted  comparative  case  studies  on  the  use  of 
information  technology  across  five  different  industries.  The  following  three 
primary  findings  of  the  research  reflect  common  arguments  among 
information  professionals. 

1.  The  need  to  define  and  track  information  technology  expenditures 
throughout  the  [organization]  is  critical,  especially  as  information 
technology  (IT)  management  becomes  more  decentralized. 

2.  The  difficulty  of  determining  the  return  on  IT  investments  is  a 
major  stumbling  block  for  many  [organizations]. 

3.  Investment  in  information  technology  alone  is  not  enough,  it  must 
link  to  business  strategy.  (Duncan,  1991:42-43) 
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Graham  claims  that  if  a  records  management  system  is  to  be  effective 
it  must  provide  tactical  information  which  details  the  critical  organizational 
functions  (1991:57).  Also,  effectiveness  in  records  management  can  be 
measured  by  lower  operating  costs,  improvement  in  the  rate  of  lost  and 
misfiled  documents,  faster  customer  service,  and  a  reduction  in  the  number 
of  personnel  required  to  run  the  program  ("Records  Management  Program...", 
1991:45).  Another  source  claims  that  effectiveness  can  be  measured  by  the 
degree  of  work  flow  improvement  and  the  reduction  of  necessary  steps  which 
are  achieved  through  functional  process  improvement  (Dykeman,  1992:54). 

The  Olsten  Corporation  has  established  an  award  for  effectiveness  in 
RM  which  clearly  outlines  the  criteria  for  judging  effectiveness.  The  areas 
evaluated  include  the  following: 

cost  savings 

space  savings 

speed  of  access  of  records 

integration  of  records  stored  on  different  formats 
management  support  of  the  RM  program 
improved  service 

labor  savings  (productivity  enhancement) 
quality  of  training 
centralization  of  records  storage 
elimination  of  lost  /  misfiled  records 
("William  Olsten  Awards  for  Excellence...,"  1993) 

As  this  section  illustrates,  the  information  industry  in  general  and 
records  managers  specifically  cannot  agree  on  what  is  required  for  an 
organization  to  be  effective  at  managing  records.  The  literature  reveals  that 
although  several  definitions  for  RM  effectiveness  have  been  proposed  the 
substance  of  any  specific  definition  is  simply  the  opinion  of  the  article's 
author;  there  is  no  industry  standard.  The  lack  of  an  accepted  standard  has 
prompted  this  thesis  research. 
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Legal  Issues 

As  advanced  technology  plays  a  bigger  part  in  handling  records, 
management  is  faced  with  critical  legal  issues.  These  issues  provide 
conspicuous  evidence  to  the  importance  of  managing  records  effectively--to 
the  necessity  of  doing  the  job  right.  Any  employment  of  new  technology  to 
accomplish  the  RM  task  must  be  evaluated  for  more  than  just  the  speed  or 
ease  it  brings  to  the  process.  To  illustrate  the  point,  federal  and  state 
governments  regulate  the  length  of  time  that  a  record  must  be  maintained 
on  file  or  in  storage.  For  example,  The  Air  Force's  Records  Information 
Management  System  (RIMS)  is  a  database  containing  in-depth  rules  for 
retention  of  official  documents.  Retention  and  destruction  schedules  are  often 
developed  to  control  record  disposal  procedures  for  an  organization 
(Sampson,  1989:80).  In  professions  such  as  insurance,  medicine,  law,  and 
finance  where  the  main  product  is  a  service,  managers  depend  heavily  on  the 
RM  process  for  proof  of  nearly  every  aspect  of  outside  transactions  (Jones, 
1993:10).  Maintenance  of  an  appropriate  disposal  policy  can  save  large  sums 
of  money  in  legal  fees  and  penalties.  Some  of  the  risks  of  inadequately 
maintaining  records  include: 

loss  of  [legal]  rights, ...  including  government  contracts,  prosecution  for 
obstruction  of  justice  if  materials  are  deliberately  destroyed  when 
litigation  or  government  investigations  are  pending  or  in  progress, 
contempt  of  court  for  deliberately  destroying  subpoenaed  evidence,  and 
presumption  of  guilt  or  other  adverse  inferences  that  come  from  not 
having  [accountable  material].  ("Records  Management  Program...", 
1991:45). 

The  threat  of  legal  action  is  as  prevalent  in  the  public  sector  as  it  is  in 
the  private  sector.  One  RM  consultant  states  "Improper  or  inadequate 
records  always  open  the  door  to  litigation, ...  preparing  complete 
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documentation  ...  is  the  most  proactive  measure  a  company  can  take.  And 
government  agencies  are  equally  obligated  ..."  (Dykeman,  1992:52).  An 
example  of  the  importance  that  some  government  agencies  are  placing  in  RM 
is  found  in  the  New  York  State  Department  of  Transportation.  Department 
managers  have  classified  RM  employee  positions  as  absolute  priorities  and 
have  funded  RM  functions  as  risk  management  activities  in  order  to  decrease 
the  rate  of  loss  due  to  litigation  (Dykeman,  1992:52). 

A  common  problem  experienced  today  arises  from  widespread  down¬ 
sizing  in  both  the  public  and  private  sectors.  RM  professionals  tend  to  be 
either  clerical  personnel  or  mid-level  managers,  each  being  a  frequent  target 
of  job  cutting  tactics.  Consequently,  RM  procedures  are  often  delegated  to 
personnel  who  are  inexperienced  and  uninterested  in  appropriate  RM 
practices,  resulting  in  misfiled,  lost,  or  improperly  disposed  of  documents 
(Jones,  1993:10-11).  For  organizations  that  fail  to  adequately  consider  their 
RM  responsibilities  the  costs  can  be  exorbitant.  For  example,  Piper  Aircraft 
lost  a  $10  million  judgment  because  of  a  failure  to  retain  a  single,  critical 
document  ("Records  Management  Program...",  1991:44).  To  prevent  such  an 
incident,  common  practice  is  for  the  records  manager  to  consult  auditors  and 
attorneys  before  establishing  retention  and  destruction  policies  ("Records 
Management  Program...",  1991:46).  Once  those  policies  are  established,  the 
RM  staff  must  adhere  to  them  rigorously  and  be  able  to  prove  in  court  exactly 
how  and  why  documents  are  destroyed.  Employment  of  computer  technology 
to  automate  the  process  should  not  only  be  weighed  on  a  cost/benefit  basis, 
but  should  also  be  carefully  configured  to  fit  the  organization’s  unique 
operating  environment  (Jones,  1993:14). 
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Delphi  Technique 

The  Delphi  method  is  a  technique  for  obtaining  a  consensus  of  a  panel 
of  experts.  The  technique  is  designed  to  capture  informed  judgment  from 
experts  and  to  provoke  candid  answers  by  avoiding  direct  confrontation 
(Brown,  1968:3).  The  experts  are  questioned  in  a  series  of  rounds  which 
allows  them  to  consider  responses  of  peers  as  they  contribute  their  own 
judgments  to  the  process  (Brown,  1968:1-3).  Objectives  of  Delphi  include 
forecasting,  qualitative  evaluations,  and  consensus  (Sackman,  1975:8).  The 
reward  for  undertaking  the  Delphi  process  is  the  development  of  a  group 
standard  which  will  advance  research  for  an  area  of  application  where  no 
expert  consensus  formerly  existed  (Sackman,  1975:6). 

The  obvious  frailty  with  the  validity  of  expert  opinion  is  finding  an 
adequate  definition  of  expert.  Brown  suggests  some  criteria: 

1)  status  among  peers 

2)  years  of  experience 

3)  self-appraisal  of  competence 

4)  quantity  of  relevant  information  the  individual  can  access 
(Brown, 1968:4). 

The  results  of  the  Delphi  process  are  usually  just  a  measurement  of 
the  experts'  perspectives  on  the  propensity  of  events  to  occur  in  the  future, 
without  actually  measuring  the  events  (Sackman,  1975:16).  Brown  sums  the 
role  and  usefulness  of  experts  as  follows: 

We  use  an  expert  because  he  has  at  his  disposal  a  large  store  of 
background  knowledge  and  a  cultivated  sensitivity  to  its  relevance 
which  permeates  his  intuitive  insight.  We  need  a  consensus  of  experts 
because  individual  experts  will  disagree  and  we  are  unwilling  to  rely  on 
the  judgment  of  a  single  specialist.  (Brown,  1968:13) 
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In  conclusion,  effectiveness  is  a  critical  issue  for  records  managers. 
Some  criteria  for  measuring  effectiveness  have  been  discussed  in  this 
chapter.  Although  no  single  effectiveness  yardstick  exists,  the  records 
manager  is  faced  with  the  challenge  of  evaluating  the  quality  of  the  process 
under  his  responsibility  to  determine  what  will  best  enhance  the 
effectiveness  of  that  process. 

Research  Design 

The  method  chosen  for  defining  effectiveness  for  RM  is  the  Delphi 
technique.  The  Delphi  technique,  as  described  earlier  in  this  chapter,  is  a 
method  of  research  designed  to  develop  a  consensus  of  opinion  among  experts 
in  a  particular  profession.  The  purpose  is  to  obtain  informed  judgment  from 
experienced  individuals  who  possess  relevant  information  for  solving 
problems  in  their  field  of  expertise  (Brown,  1968:3-4).  A  successful  Delphi 
will  develop  a  standard  or  measure  which  will  advance  research  in  an  area 
where  no  consensus  existed  previously  (Sackman,  1975:6).  This  research  will 
attempt  to  do  just  that-attain  a  consensus  from  experts  in  an  effort  to  define 
effectiveness  for  the  field  of  RM,  where  no  such  consensus  exists. 

Delphi  Methodology 

The  first  step  in  the  Delphi  is  to  choose  the  experts  who  will 
participate.  The  group  will  consist  of  former  graduates  from  the  Air  Force 
Institute  of  Technology  (AFIT)  in  the  discipline  of  Information  Resources 
Management  (IRM).  These  participants  are  defined  as  experts  because  of  two 
major  criteria.  First,  they  all  have  at  least  four  years  of  experience  within  the 
RM  environment  in  positions  of  responsibility  and  leadership  as  both 
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enforcers  of  RM  policy  and  as  innovators  in  the  field.  Second,  they  are  the 
functional  RM  experts  in  the  Air  Force  because  of  their  graduate  education 
credentials. 

To  obtain  expert  participants  in  the  Delphi,  the  research  team  will 
send  a  request  for  participation  to  all  Air  Force  officers  who  qualify  as 
experts  as  described  above.  The  actual  number  of  participants  will  depend  on 
the  number  of  responses  received  from  individuals  willing  to  participate. 

Conducting  a  Delphi  involves  an  iterative  process  incorporating 
several  rounds  of  questioning  and  consensus  building  tactics  (Brown, 

1968:3).  The  first  round  will  consist  of  questions  developed  from  the  results  of 
a  Functional  Process  Improvement  (FPI)  process  conducted  at  the  Pentagon 
for  the  Office  of  the  Secretary  of  the  Air  Force.  The  FPI  contained  an 
evaluation  of  RM  effectiveness  criteria  cultivated  from  exhaustive  meetings 
of  RM  professionals  Air  Force  wide.  Results  from  the  FPI  will  be  used 
because  they  represent  the  only  benchmark  for  RM  effectiveness  that  is 
available  with  unique  Air  Force  pertinence.  Furthermore,  the  FPI  employed 
a  consensus-gathering  method  to  develop  effectiveness  criteria  among 
individuals  who  have  some  degree  of  knowledge  and  experience  in  RM. 

These  effectiveness  criteria  will  form  the  basis  of  questioning  for  the  Delphi 
panel  in  an  attempt  to  either  validate  the  results  of  the  FPI  or  to  create  new 
and  more  precise  measures  of  effectiveness  using  experts. 

The  Delphi  will  be  conducted  in  two  phases.  The  first  phase  will  be  a 
rating  of  the  importance  of  each  function  being  considered.  Delphi  panel 
members  will  be  asked  to  rate  the  importance  of  the  issue  stated  with  a 
number  from  one  to  seven,  one  being  very  unimportant  and  seven  being  very 
important.  The  mean  and  range  will  be  obtained  from  the  responses  of  each 
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statement  with  the  results  and  the  original  statements  returned  to  the 
participants  in  any  necessary  subsequent  rounds.  This  will  provide  the  panel 
members  with  the  opportunity  to  reconsider  their  original  responses  in  light 
of  the  responses  of  the  entire  panel.  Any  change  that  a  participant  decides  to 
make  to  his  or  her  original  response  will  be  done  at  this  point.  The  purpose  is 
to  bring  the  panel  closer  to  reaching  consensus-by  being  allowed  to  reflect  on 
the  responses  of  the  group  as  a  whole,  they  can  reevaluate  their  own  opinions 
and  make  any  appropriate  modifications  to  their  original  answers. 

The  second  phase  will  be  a  rank  ordering  of  those  functions  to 
determine  their  relative  importance.  The  same  nine  statements  included  in 
the  first  phase  rounds  will  be  used.  The  participants  will  then  be  asked  to 
provide  their  opinions  of  the  order  of  importance  of  the  issues,  ranking  them 
from  one  to  nine  with  one  being  the  most  important  and  nine  being  the  least 
important.  As  in  phase  one,  the  panel  members  will  be  given  the  opportunity 
to  reevaluate  their  responses  and  make  any  desired  changes  in  later  rounds. 
This  will  be  done  in  an  effort  to  provide  closer  agreement  among  the  experts. 

The  phase  1  instructions  to  be  provided  to  the  participants  and  a 
description  of  the  records  management  functions  proposed  for  consideration 
are  reproduced  in  Appendix  A.  The  phase  2  instructions  are  reproduced  in 
Appendix  B. 

The  purpose  of  the  Delphi  is  to  generate  information  which  could  be 
used  by  managers  to  help  them  in  developing  information  systems  for  their 
organizations.  An  expert-generated  list  of  critical  issues  ranked  in  order  of 
importance  is  seen  by  the  researchers  as  helpful  to  managers  in  decision 
making. 
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Once  the  panel  has  reached  consensus  regarding  the  ranking  of  the 
issues  provided  to  them,  the  Delphi  will  be  complete.  At  that  point,  the 
research  team  will  have  developed  an  expert  consensus  regarding  the  most 
important  issues  to  be  addressed  when  considering  a  purchase  of  a  system  for 
managing  RM--a  definition  of  what  a  system  must  be  able  to  accomplish  in 
order  to  be  effective  at  managing  records. 

Delphi  Results 

This  section  describes  the  data  collected  by  administering  the  Delphi 
process.  The  Delphi  results  are  summarized,  showing  how  consensus  was 
reached,  and  sample  instructions  are  given  to  show  how  the  process  evolved. 

Participants 

Forty-eight  potential  Delphi  participants  were  solicited,  thirteen 
actually  agreed  to  be  panel  members.  All  participants  were  graduates  of  the 
AFIT  IRM  program,  and  possessed  at  least  four  years  of  experience  with  the 
RM  field. 

Round  One  Results  and  Comments 

The  results  of  round  one  and  round  two  are  listed  in  Appendix  C.  The 
various  functions  rated  represent  the  rows,  and  the  rating  scale  of  one  to 
seven  is  reproduced  to  create  the  columns  of  the  table.  The  data  represent 
the  frequency  that  a  particular  score  was  chosen  for  a  particular  function. 
Descriptive  statistics  appear  in  Table  2  for  comparison  with  round  two. 
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Round  One  Feedback  and  Comments 


The  following  text  lists  the  numerical  results  of  round  one  that  were 
given  to  the  participants  for  a  second  round.  It  is  included  here  because  the 
bulk  of  comments  received  appeared  in  the  first  round.  Information  given 
included  the  range  of  values,  the  most  common  value  (mode),  any  exceptions 
to  the  range  (single  occurrence  deviations  were  considered  exceptions), 
comments  related  to  the  exceptions,  and  general  comments  or  comments 
unrelated  to  the  exceptions  for  each  of  the  nine  functions.  Individual  scores 
were  not  returned  to  the  participants  due  to  an  administrative  oversight. 


a.  Filtering  information 

Range  of  responses:  4  to  7 

Most  common  rating:  5 

Exception(s):  2 

Comments  related  to  exceptions: 

Filtering  information  needs  to  be  a  manual  process  done 
by  the  owner  of  the  documentation.  I  don't  need  the  machine  to 
tell  me  whether  an  item  should  be  put  into  the  records 
management  system.  I  just  need  it  to  be  simple  to  add  records  I 
do  want  to  file. 

Comments  unrelated  to  exceptions:  None 

b.  Making  records 

Range  of  responses:  5  to  6 

Most  common  rating:  6 

Exception(s):  3,7 

Comments  related  to  exceptions:  None 
Comments  unrelated  to  exceptions:  None 

c.  Assigning  disposition  instructions  automatically 

Range  of  responses:  5  to  7 

Most  common  rating:  5 

Exception(s):  2,3,4 
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Comments  related  to  exceptions: 

Please  remember  that  BASE  X  is  drawing  down.  We  are  very 
concerned  about  the  transfer  and  destruction  of  records  (h  &  i). 
In  the  overall  picture  of  an  operational  base,  people  don't  pay  as 
much  attention  to  the  eventual  disposition  of  the  records  as  they 
should.  Therefore,  it  is  up  to  us  to  find  and  correct  years  worth 
of  mistakes  before  disposing  of  records  [editors  note:  seems  to  be 
related  to  c  but  was  reported  with  miscellaneous]. 

Comments  unrelated  to  exceptions:  None 

d.  Indexing  records  automatically 

Range  of  responses:  4  to  7 

Most  common  rating:  6 

Exception(s): 

Comments  related  to  exceptions:  None 
Comments  unrelated  to  exceptions:  None 

e.  Storing  records 

Range  of  responses:  6  to  7 

Most  common  rating:  7 

Exception(s): 

Comments  related  to  exceptions:  None 
Comments  unrelated  to  exceptions:  None 

f.  Screening  records 

Range  of  responses:  6  to  7 

Most  common  rating:  6 

Exception(s):  3 

Comments  related  to  exceptions:  None 
Comments  unrelated  to  exceptions:  None 

g.  Retrieving  records 

Range  of  responses:  4  to  7 

Most  common  rating:  7 

Exception(s):  3 

Comments  related  to  exceptions: 

Ease  of  retrieval  has  NOT  always  been  a  prime  concern  of 
NARA  (officially,  it  was;  but  practically,  is  was  NOT).  With 
digital  records,  retrieval  issues  should  be  easily  addressable  up- 
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front:  Coding  (retention)  schemes  and  standards  must  be 
ongoing  concerns  in  this  area. 

Ultimate  security  of  federal  records  is  of  paramount 
importance,  and  this  will  continue  to  be  a  prime  objective  of  the 
federal  records  system. 

Comments  unrelated  to  exceptions: 

NOTE:  Isn't  this  really  two  different  questions,  one 
dealing  with  locating  records  and  the  other  dealing  with  access 
rights  and  security?  These  are  two  different  activities. 

h.  Allowing  the  transfer  of  records  to  permanent  storage  sites 
and  maintaining  a  detailed  audit  trail 

Range  of  responses:  5  to  7 

Most  common  rating:  6 

Exception(s):  4 

Comments  related  to  exceptions:  None 
Comments  unrelated  to  exceptions: 

Another  two-part  question.  Transferring  records  from 
one  location  to  another  and  auditing  those  transfers  are  two 
entirely  different  activities. 

i.  Destroying  records  automatically,  in  accordance  with 
prescribed  retention  directives 

Range  of  responses:  5  to  7 

Most  common  rating:  5 

Exception(s):  1,3 

Comments  related  to  exceptions: 

Automatically  queuing  records  for  human  review  would 
be  a  7.  I  don't  want  the  system  automatically  destroying 
anything  without  someone  checking  the  destruction  queue  first, 
though. 

Comments  unrelated  to  exceptions: 

Destruction  of  records  should  be  a  task  that  you  are 
prompted  to  do  on  a  set  interval  (monthly?)  so  you  are  not 
continuously  confronted  with  records  to  be  destroyed. 

Automatic  destruction  of  records  should  not  be  done  without 
notification. 

Retention  of  records  is  (or  at  least  has  historically  been) 
based  on  the  value  (or  evident  potential  value)  of  the 
information  contained.  This  is  reflected  in  disposition  codes. 
These  codes  are  placed  on  documents/records  either  manually, 
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or  through  some  use  of  appropriate  automation;  but  the 
rationale  is  always  dependent  on  human  interpretation.  Thus  I 
would  NOT  want  to  see  any  system  which  would 
AUTOMATICALLY  destroy/dispose  of  records  without  a  second 
look.  I  would  want  to  see  a  system  that  would  alert  both  the 
record  owner  and  the  resident  records  management  expert  that 
a  record  is  due  for  ultimate  disposal. 

Other  miscellaneous  comments: 

Hopefully  your  thesis  team  is  talking  with  Darryl 
Prescott  at  SAF/AAIXI.  He  feels  (  and  I  agree)  that  6000  tables 
and  rules  for  the  disposition  of  official  records  is  INSANE!! 

I'm  also  sure  you're  looking  at  the  Document  Librarian 
program  coming  out  of  AFMC.  We  re  looking  at  implementing  it 
here  (it  would  be  more  correct  to  say  we'll  implement  unless 
someone  tells  us  not  to!!). 


Round  Two  Results  and  Comments 

The  results  of  round  two  are  listed  in  Appendix  C  for  simplicity  in 
comparing  round  results.  No  significant  comments  were  received.  Although 
the  round  one  scores  were  not  provided  as  feedback,  the  consensus  got  better 
as  demonstrated  by  the  means  and  standard  deviations  in  the  following 
table. 
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Table  1 

DESCRIPTIVE  STATISTICS  FOR  ROUNDS  ONE  AND  TWO 


Function 

Round 

AVG 

STDev 

a.  filtering 

rl 

5.15 

1.34 

r2 

5.00 

1.35 

b.  making 

rl 

5.54 

0.97 

r2 

5.69 

0.48 

c.  disposition 

rl 

5.46 

1.56 

r2 

5.62 

1.04 

d.  indexing 

rl 

5.77 

1.01 

r2 

5.92 

0.64 

e.  storing 

rl 

6.85 

0.38 

r2 

.  6.92 

0.28 

f.  screening 

rl 

6.31 

0.48 

r2 

6.31 

0.48 

rl 

5.85 

1.41 

r2 

6.08 

1.26 

h.  transfer  to  storage 

rl 

5.85 

0.80 

r2 

5.92 

0.64 

rl 

5.23 

1.74 

r2 

4.62 

1.56 

Round  Three  Instructions 

Satisfied  with  the  results  of  rounds  one  and  two  at  reaching  consensus 
on  rating  of  the  functions,  we  shifted  emphasis  to  a  ranking  of  the  functions 
in  an  attempt  to  cross-check  the  opinions  of  our  experts.  This  method  had 
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also  been  specifically  discussed  with  our  sponsor  in  an  effort  to  produce 
weighting  factors  for  evaluation  of  future  records  management  systems, 
whether  they  would  be  research  and  development  projects  or  commercial  off- 
the-shelf  products. 

Round  Three  Results  and  Comments 

Appendix  D  shows  round  three  and  four  results  in  a  format  similar  to 
Appendix  C.  The  rows  represent  functions  to  be  ranked  and  the  columns  the 
frequency  of  choice  of  each  rank.  Descriptive  statistics  appear  in  Table  2. 

Round  Three  Feedback/Prenaration  of  Round  Four 

The  results  from  round  three  were  given  back  to  the  participants  in 
summary  form,  including: 

-  previous  ranking  for  each  individual  (given  privately  and 
accomplished  at  email  time) 

-  each  function  and  all  ranks  selected  for  it 

-  mean  and  standard  deviation  of  those  ranks 

*  NOTE  that  the  previous  round's  ranking  for  each  individual  was 
included  as  feedback. 

Round  Four  Results  and  Comments 

The  results  of  round  four  are  listed  in  Appendix  D.  Descriptive 
statistics  are  provided  in  table  2  for  simplicity  in  comparing  round  results. 
No  significant  comments  were  received. 
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Table  2 


DESCRIPTIVE  STATISTICS  FOR  ROUNDS  THREE  AND  FOUR 


Function 

Round 

AVG 

STDev 

a.  filtering 

r3 

7.54 

1.81 

r4 

7.54 

1.71 

b.  making 

r3 

5.69 

2.21 

r4 

5.38 

1.94 

c.  disposition 

r3 

5.00 

1.68 

r4 

4.69 

1.38 

d.  indexing 

r3 

5.08 

2.14 

r4 

4.62 

0.65 

e.  storing 

r3 

1.62 

1.94 

r4 

1.62 

1.94 

f.  screening 

r3 

3.15 

1.91 

r4 

3.08 

2.25 

g.  retrieving 

r3 

3.46 

1.66 

r4 

3.31 

1.60 

h.  transfer  to  storage 

r3 

5.92 

1.55 

r4 

6.77 

1.64 

i.  destroying 

r3 

7.54 

1.66 

r4 

8.00 

1.35 

Analysis  of  Results 

For  purposes  of  discussion,  we  will  call  rounds  one  and  two  the  rating 
or  first  stage  and  rounds  three  and  four  the  ranking  or  second  stage. 

Feedback  data.  At  least  one  clear  pattern  emerged  to  distinguish  a 
given  function  as  universally  important  to  all  participants.  Function  E, 
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storing  records,  had  the  highest  mean  of  over  6.9  in  the  rating  stage  and  was 
almost  unanimously  ranked  number  one.  The  exception  was  the  outlier  case; 
a  single  participant  who  often  disagreed  with  the  others  and  never  changed 
his  mind  between  rounds.  This  person's  choices  were  not  considered 
significant  enough  to  interfere  with  consensus,  however,  so  they  remain  in 
the  data  set. 

In  the  rating  stage  all  of  the  functions  were  considered  important  to 
the  process,  and  only  one  mean  in  either  round  was  below  5.0.  That  one 
mean  belonged  to  the  clear  loser,  however,  as  the  ranking  stage  gave  it  a 
clear  ninth  place  and  earlier  comments  disclosed  that  Function  I,  destroying 
records  (in  an  automatic  fashion),  was  unwelcome  in  any  system. 

Consensus.  If  no  progress  or  negative  progress  had  been  reached 
toward  consensus  in  each  round,  we  would  have  stopped  the  process  at  that 
point.  Since  progress  was  made  with  the  majority  of  our  functions  in  each  of 
our  rounds  (as  measured  by  a  decreasing  standard  deviation),  we  continued 
until  the  extent  of  time  available  was  reached  and  declared  closure  based  on 
the  indication  of  the  thesis  advisors. 
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III.  THE  IMPACT  OF  DOCUMENT  LIBRARIAN 


ON  RECORDS  MANAGEMENT 

This  chapter  includes  discussions  on  new  advances  in  the  area  of 
electronic  imaging  and  then  some  examples  of  the  successes  and  failures 
resulting  from  the  implementation  of  information  technology.  These 
examples  highlight  the  trends  in  RM  that  justify  the  need  for  further 
research  regarding  the  impact  that  electronic  technology  has  on  the  records 
management  process.  The  remainder  of  the  chapter  contains  the  methodology 
for  answering  the  second  investigative  question,  and  the  results  of  carrying 
out  that  methodology. 

Electronic  Imaging 

A  significant  although  quite  new  addition  to  the  tools  of  records 
management  is  electronic  document  imaging.  In  contrast  to  photographic 
imaging  (microfilm),  an  electronic  imaging  device  scans  a  document  and 
creates  a  bit  map  for  storage  on  magnetic  or  optical  disks  (Shaw,  1993:10). 
The  chief  benefit  of  electronic  over  photographic  imaging  is  the  rapid 
retrieval  facilitated  by  the  electronic  storage  medium.  An  image  can  be 
retrieved  in  seconds  which  is  a  dramatic  improvement  over  the  often  lengthy 
searches  necessary  for  microfilm  (Shaw,  1993:10). 

Another  benefit  is  the  capacity  to  share  access  to  documents  across  a 
wide  band  of  users  (Corbin,  1992:20).  If  a  record  must  be  processed  by 
workers  in  different  locations,  each  has  ready  access  to  the  record  which  can 
sometimes  eliminate  days  of  transit  time  (Shaw,  1993:10).  Government 
organizations  can  particularly  benefit  from  this  capacity  due  to  the  method 
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by  which  much  of  the  work  is  accomplished.  Often  employees  at  several 
levels  of  management  or  in  different  areas  of  responsibility  review  and  share 
documents.  When  such  documents  can  be  passed  from  one  work  station  to 
another  in  a  matter  of  seconds  using  electronic  imaging,  the  opportunity  to 
improve  productivity  is  attractive. 

The  principal  obstacle  preventing  the  wide  use  of  electronic  imaging  to 
route  work  flows  has  historically  been  the  requirement  of  additional 
purchases  of  hardware  to  facilitate  the  application.  The  costs  of  bringing 
imaging  systems  on  line  has  commonly  run  between  $50,000  and  $500,000 
(Corbin,  1992:20).  Only  in  recent  years  have  software  engineers  been  able  to 
accommodate  users  with  products  that  can  be  installed  on  computer  systems 
already  in  place  (Shaw,  1993:12).  Consequently,  the  use  of  imaging  to  route 
work  flows  is  on  the  increase.  Several  commercial  vendors  now  market  work 
flow  software  packages  that,  with  the  attention  of  a  systems  administrator, 
can  be  uniquely  tailored  to  an  organization's  needs  at  dramatically  reduced 
costs  (Shaw,  1993:10). 

Still,  the  main  use  of  electronic  imaging  is  to  store  records  on  optical 
disk  to  eliminate  the  need  to  store  large  quantities  of  paper  files.  The  Air 
Force  Military  Personnel  Center  at  Randolph  Air  Force  Base  spent  $5.8 
million  in  1992  for  an  imaging  system  which  will  be  used  for  storage  and 
retrieval  of  thousands  of  personnel  records  from  around  the  Air  Force 
(Corbin,  1992:21).  Most  organizations  require  some  degree  of  retention  of 
historical  data  as  a  matter  of  policy  to  satisfy  both  legal  and  operational 
concerns.  Imaging  is  popular  for  this  purpose  because  of  its  efficiency  in 
storing  and  retrieving  documents  ("DMI  Has  Solution...",  1993:48).  Other 
commonly  used  long  term  storage  mediums  such  as  magnetic  tape,  microfilm, 
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and  microfiche  have  consistently  proven  to  be  inefficient  and  somewhat 
insecure  methods  of  managing  historical  data  ("DMI  Has  Solution...”, 
1993:48).  Optical  disks,  which  are  the  standard  medium  used  with  imaging 
for  long  term  storage,  provide  easier  and  faster  retrieval  of  records,  as  well  as 
the  additional  benefit  of  security  since  the  disks  cannot  be  altered  ("DMI  Has 
Solution...",  1993:48). 

Despite  market  trends,  opinions  vary  widely  as  to  the  contributions 
that  electronic  imaging  can  make  to  the  effectiveness  of  records  management. 
The  early  users  of  imaging  equipment  experienced  some  significant 
disappointments  which  could  have  soured  RM  professionals  on  the 
technology.  However,  the  consensus  is  that  imaging  will  have  a  noticeably 
positive  impact  on  the  dependency  on  and  proliferation  of  paper,  the  speed  of 
access  of  information,  and  the  cost  of  physically  storing  documents  (Corbin, 
1992:20). 

Another  key  issue  is  whether  electronically  reproduced  documents  will 
stand  up  as  evidence  in  court.  No  clear  precedent  exists  to  serve  as  a 
definitive  guide  to  management  (Boudette,  1990:64).  Microfilm  is  admissible 
as  evidence  under  the  guidelines  of  the  Uniform  Photographic  Copies  of 
Business  and  Public  Records  Act  and  most  federal  managers  feel  that  the 
courts  will  grant  electronically  reproduced  documents  the  same  weight 
(Sampson,  1989:80).  This  assumption,  however,  has  not  yet  been  tested  at  all 
levels  of  the  judicial  system.  There  are  some  federal  and  state  statutes  that 
mandate  certain  records  being  maintained  in  paper  form  for  a  specified 
period  of  time  or  until  they  are  obsolete  (Sampson,  1989:80).  Furthermore, 
electronic  data  is  at  present  not  generally  accepted  as  judicial  evidence 
(Sampson,  1989:80). 
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Successful  Implementation  of  Technology 


Because  the  purpose  of  this  study  is  to  evaluate  the  impact  of 
technology  on  the  effectiveness  of  records  management,  information  about 
successes  and  failures  of  technology  implementation  is  necessary  to  provide 
justification  for  research.  Many  examples  of  each  exist:  this  section  deals 
with  successes  found  primarily  in  the  public  sector. 

The  Social  Security  Administration,  in  response  to  intense  pressure  to 
cut  costs,  developed  a  new  program  to  modernize  the  agency's  business 
processes.  The  agency  integrated  commercial  telecommunications  lines  with 
updated  computer  technology  in  an  effort  to  reduce  costs  and  improve 
customer  service  (McDonough  and  Buckholtz,  1992:34).  Customer  inquiries 
that  were  previously  done  on  paper  could  then  be  done  by  computer.  The 
payoff  has  been  a  reduction  of  17,000  positions  along  with  impressive 
reductions  in  the  length  of  service  delivery  time  (see  Table  3).  In  addition  to 
reduced  customer  service  delivery  time,  the  agency  has  also  realized  an 
excellent  improvement  in  the  rate  of  payment  errors;  one  county's  error  rate 
dropped  by  more  than  85  percent  (McDonough  and  Buckholtz,  1992:34). 


Table  3 

SERVICE  DELIVERY  IMPROVEMENTS  OF  THE  SSA 
(McDonough  and  Buckholtz,  1992:34) 


Seryiqq 

1982 

1991 

Card  Receipt 

6  weeks 

3-5  days 

Annual  Wage  Postings 

4  years 

6  months 

Cost  of  Living  Adjustments 

3  weeks 

1  day 

Emergency  Payments 

15  days 

3-5  days 
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Several  federal  agencies  have  implemented  pilot  projects  using 
electronic  imaging  systems  with  dramatic  results.  Productivity  has  improved, 
operations  have  been  streamlined,  customer  service  has  improved,  and  entire 
functional  processes  have  been  curtailed  (Boudette,  1990:61-62).  As  one 
noteworthy  illustration,  the  Federal  Deposit  Insurance  Corporation  (FDIC) 
used  a  new  optical  disk  system  to  replace  an  outdated  system  of  paper  files 
and  microfiche.  Using  the  new  storage  medium  has  improved  both  the  FDIC's 
service  to  its  member  banks  and  the  service  that  banks  provide  to  the  general 
public.  The  improvements  include  a  reduction  in  the  amount  of  time  that 
staff  spends  on  research,  improved  security  of  the  stored  data,  and  such  a 
large  reduction  in  the  quantity  of  paper  generated  that  the  paper  documents 
which  could  fill  a  space  estimated  at  8'x35'x50'  are  condensed  to  50  optical 
disks  (Arend,  1993:55). 

A  conspicuous  example  at  the  state  level  of  new-found  success  in  the 
RM  field  has  occurred  at  the  Kansas  Department  of  Health  and  Environment 
(KDHE).  The  agency  is  responsible  for  maintaining  over  eight  million  records 
for  the  state  and  performs  approximately  350,000  transactions  with  the 
public  each  year  ("Records  Management  Program...",  1991:50).  Two  main 
problems  prompted  KDHE  to  develop  a  new  RM  system:  1)  response  time  to 
customer  requests  for  copies  of  records  was  3-5  weeks  and  2)  the  excessive 
quantity  of  film  and  paper  records  required  the  use  of  five  different  locations 
with  under-ground  vaults  for  storage  ("Records  Management  Program...", 
1991:50). 

After  three  years  of  investigation  of  legal  and  accounting  requirements 
as  well  as  budgetary  fights,  the  new  system  was  implemented.  The  final 
product  integrated  an  IBM  mainframe  database  with  an  NCR  minicomputer 
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and  a  FileNet  image  system  ("Records  Management  Program...",  1991:50). 
Several  benefits  were  quickly  realized  including  a  reduction  of  transaction 
error  rate  of  30  percent  to  less  than  4  percent,  a  virtual  elimination  of  data 
entry  by  state  personnel,  and  a  reduction  in  the  customer  request  response 
time  from  3-5  weeks  to  as  little  as  11  seconds  ("Records  Management 
Program...",  1991:50). 

Failures  of  Technology  to  Improve  Effectiveness 

The  growth  in  the  availability  of  information  technology  has  attracted 
the  interest  of  many  managers  seeking  to  improve  the  financial  and 
functional  performance  of  their  areas  of  responsibility.  However,  when 
managers  who  lack  experience  and  knowledge  of  information  technology 
make  purchasing  decisions,  the  organization  can  quickly  run  into  unforeseen 
problems  (Corbin,  1992:41).  The  General  Accounting  Office  in  1992  reported 
that  many  information  technology  systems  throughout  the  federal 
government  were  not  only  inadequate  for  their  intended  functions,  but  were 
so  poorly  managed  that  many  bad  situations  were  made  worse  (Corbin, 
1992:44).  Nolan  Norton  Industries  investigated  600  imaging  test  projects  and 
collected  very  negative  data.  Nolan  found  that  17.3%  of  the  new  projects  had 
positive  progress,  2.1%  had  negative  progress,  and  80.6%  had  no  progress 
(Booker,  1990:89).  However,  one  industry  leader  at  International  Data  Corp. 
(IDC),  in  Framingham,  MA  claims  that  the  large  number  of  firms  showing  no 
progress  simply  indicates  poor  management  of  the  technology  and 
information  systems  in  general.  IDC  conducted  an  independent  study  of  225 
on-site  imaging  systems  and  found  only  14  percent  to  have  experienced  any 
problems  at  all  (Booker,  1990:89).  IDC  goes  further  to  claim  that  most  of  the 
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problems  that  do  occur  are  caused  by  managers  who  attempt  to  automate 
processes  which  aren't  fully  understood  by  the  organization  (Booker, 

1990:89). 

The  California  secretary  of  state's  office  in  March  of  1989  installed  a 
$4  million  system  from  FileNet  Corp.  in  an  effort  to  improve  the  management 
of  commercial  filings.  The  outcome  is  now  infamous  within  the  information 
management  profession.  The  contract  was  for  an  integrated  system  of 
computing  and  imaging,  but  problems  came  about  immediately  after 
implementation.  The  department  was  never  able  to  accomplish  the  required 
amount  of  work  within  any  single  day.  The  most  successful  production  level 
ever  achieved  while  using  the  new  system  was  to  process  30  percent  of  the 
workload  demanded  (Booker,  1990:89).  Predictably,  the  customers  of  the 
Secretary  of  State's  office  were  highly  displeased  about  what  ultimately 
became  a  two-month  backlog  of  work.  This  forced  the  state  to  remove  the 
system  and  employ  250  extra  people  working  around  the  clock  to  catch  up  on 
the  backlog  (Booker,  1990:89). 

The  field  of  information  technology  in  general  has  come  under  fire 
from  managers  for  its  apparent  inability  to  bear  fruit  in  the  area  of 
productivity.  In  the  1980' s,  productivity  in  the  manufacturing  sector 
increased  44  percent,  but  in  the  service  sector,  where  85  percent  of  IT 
spending  occurs,  productivity  increased  less  than  2  percent  (Schnitt, 

1993:15).  The  Massachusetts  Institute  of  Technology  (MIT)  studied  this  data 
and  determined  that  "'Expenditures  on  IT  capital  were  less  effective  in 
improving  productivity  than  any  other  type  of  expenditure  considered'" 
(Schnitt,  1993:15).  Most  troubling  for  the  IT  industry  was  the  finding  that 
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the  most  productive  organizations  studied  committed  less  capital  on  IT  than 
those  organizations  with  average  productivity  (Schnitt,  1993:15). 

Schnitt  suggests  that  the  reason  for  the  general  lack  of  productivity 
improvement  is  the  tendency  of  workers  to  concentrate  on  the  technology 
itself  rather  than  what  the  technology  can  do  for  the  organization  (Schnitt, 
1993:15).  IT  professionals  frequently  design  computer  systems  to  fit  into 
existing  work  processes  without  questioning  whether  the  processes 
themselves  are  the  most  productive  method  of  performing  the  work.  To 
illustrate,  the  increase  of  productivity  in  the  manufacturing  sector  is  credited 
to  the  increase  in  foreign  competition,  which  is  greater  for  manufacturers 
than  for  the  service  sector.  The  competition  forces  manufacturers  to  improve 
productivity  and  in  turn  compels  them  to  use  investments  in  IT  to  make 
changes  in  the  way  workers  perform  their  jobs.  This  is  what  is  lacking  in  the 
service  sector  (Schnitt,  1993:15-16). 

In  today's  working  environment,  with  the  pressures  of  budgets  as 
severe  as  they  are  in  government  and  in  business,  managers  must  be  as 
certain  as  possible  that  information  technology  will  actually  improve  their 
RM  activities  before  committing  the  capital  to  acquire  that  technology. 

The  previous  sections  of  this  chapter  comprise  a  discussion  on  how 
electronic  technology  affects  the  RM  process.  The  following  section  is  a 
description  of  the  method  to  be  used  in  evaluating  the  unique  impact  of 
Document  Librarian  on  that  process. 

Survey  Methodology 

The  sponsor  of  this  research,  AFMC  Corporate  Information  (Cl),  has 
introduced  into  AFMC  headquarters  the  commercial  RM  software  product, 
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Document  Librarian  (DL).  This  study  will  incorporate  a  customer  satisfaction 
survey  of  the  population  which  is  testing  the  new  product. 

Emory  and  Cooper  state  that  the  most  appropriate  employment  of  a 
survey  is  with  a  population  "where  conditions  indicate  that  respondents  are 
uniquely  qualified  to  provide  the  desired  information"  (1991:319).  The 
population  testing  DL  is  the  principal  user  group  of  the  software.  Not  only 
are  they  experienced  with  the  product  but  they  also  have  a  vested  interest  in 
its  effective  performance. 

The  type  of  survey  to  be  used  is  an  evaluation  survey  as  suggested  by 
Fox  (1969:434).  This  is  most  suitable  because  an  evaluative  judgment  can  be 
made  in  a  situation  with  a  criterion  measure  (user  satisfaction),  and  the 
survey  can  be  carried  out  using  a  single  test  group  (Fox,  1969:434).  Fox  also 
states  that  an  evaluation  survey  is  particularly  suited  to  measure  a  change 
over  time  (1969:436).  Since  this  research  seeks  to  identify  how  well  the  users 
of  the  RM  software  are  satisfied  with  the  product,  the  perceived  change  in 
RM  effectiveness  on  the  part  of  the  test  population  occurring  after  DL's 
implementation  is  what  will  be  measured. 

Survey  Instrument.  Section  1  of  the  survey  will  gather  demographic 
data  on  the  population.  Section  2  will  relate  to  satisfaction  with  the  various 
aspects  (from  security  to  convenience)  of  DL.  Finally,  section  3  will  deal  with 
an  overall  impression  of  its  user  'feel'  (from  the  user  interface  to  ease  of 
learning).  Questions  in  sections  2  and  3  will  be  a  rating  on  a  scale  from  1  to 
7  with  a  value  of  1  being  unfavorable  and  a  value  of  7  being  favorable.  A 
value  of  4  will  represent  a  neutral  position.  For  the  convenience  of  the 
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reader,  the  survey  questions,  in  the  form  presented  electronically  to  the 
participants,  are  reproduced  in  Appendix  E  for  reference. 

Instrument  Validation.  The  source  of  the  instrument  is  a  master's 
thesis  written  by  Randall  R.  Bradford  (1993).  Bradford  tested  the  validity  of 
the  instrument  and  reported  a  Cronbach's  Alpha  of  .96,  indicating  that  the 
instrument  is  internally  consistent.  Since  the  source  questionnaire  was 
tailored  to  an  electronic  mail  system,  this  thesis  team  will  make  minor 
modifications  to  the  questions  to  tailor  the  instrument  for  the  DL  application. 
Also,  one  question  in  the  source  instrument  which  appears  to  duplicate 
another  question  will  be  eliminated  in  this  instrument.  The  numerical  ranges 
on  all  demographic  items  will  be  changed  to  make  them  mutually  exclusive. 

The  altered  survey  instrument  will  be  validated  prior  to  being 
submitted  to  the  test  population.  The  purpose  is  to  test  whether  the  questions 
asked  are  clear  and  that  respondents  understand  the  intent  of  the  question. 
The  population  which  will  be  used  for  validating  is  10  members  from 
AFMC/CI,  who  are  the  sponsors  of  this  research.  They  will  be  given  the 
survey  and  asked  to  suggest  changes  in  the  wording  and  the  structure  of  the 
instrument  that  would  make  it  easier  to  understand  and  answer.  Once  this 
process  is  complete,  the  final  survey  will  then  be  submitted  to  the  test 
population. 

Analysis  of  Survey  Results.  The  data  collection  process  will  begin  with 
the  generation  of  the  survey  instrument.  The  survey  to  be  used  was  design  to 
gather  data  about  both  user  demographics  and  satisfaction  with  the  product. 
The  questions  will  be  tested  initially  to  evaluate  their  content  validity. 
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To  convert  data  into  useful  information,  the  team  will  analyze  the 
survey  results  in  several  areas.  The  first  area  of  interest  will  be  the 
demographic  responses.  A  compilation  of  this  data  will  provide  the  sponsor 
with  more  detailed  information  about  the  population.  The  next  area  of 
analysis  will  be  testing  the  correlation  of  each  question  with  the  whole  of  the 
survey.  This  information  will  indicate  the  degree  to  which  any  single 
question  impacts  the  overall  results--how  much  the  response  for  a  question 
affects  the  responses  to  other  questions. 

In  an  effort  to  evaluate  the  satisfaction  with  DL  within  the  population, 
the  research  team  will  begin  with  the  mean  of  responses  to  individual 
questions  to  see  if  respondents  are  satisfied  or  dissatisfied  with  specific  areas 
of  the  software  application.  The  team  will  also  break  out  responses  by 
demographics  to  determine  if  a  trend  exists  with  regard  to  what  type  of 
person  (demographically)  may  have  responded  to  a  question  in  a  particular 
way.  Finally,  the  team  will  examine  areas  of  satisfaction  with  unusually  high 
or  low  ratings.  This  data  should  provide  information  about  the  definitive 
strengths  or  weaknesses  with  the  DL  software. 

Administration 

The  following  sections  contain  the  steps  taken  to  implement  the  survey 
methodology  described  above. 

Electronic  Survey  Methodology.  Considerable  effort  was  spent  in 
developing  a  method  for  surveying  users  electronically  using  software  tools 
available  to  every  user  at  AFMC.  These  instructions  were  thought  by  our 
sponsor  to  be  worthy  of  a  separate  appendix  and  are  attached  as  Appendix  F. 


32 


In  the  wake  of  our  growing  environment  of  "quality  awareness"  many  brand 
new  researchers  will  be  tasked  with  surveying  their  customer's  needs  and 
doing  so  quickly.  This  method  could  be  adapted  quite  easily  to  meet  their 
needs. 


Survey  Instructions.  The  instructions  given  with  the  survey  mainly 
dealt  with  the  complexity  of  answering  the  questions  using  the  combination 
of  electronic  mail  and  electronic  forms.  In  addition  to  a  statement  regarding 
the  authority  for  the  survey  itself,  we  included  a  step-by-step  instruction  set 
including  every  prompt  the  user  would  encounter,  and  the  actions  to  be 
taken.  This  was  by  far  the  largest  part  of  the  effort  and  took  several  rounds 
of  modification. 

Pre-Test  Results-Survev  Instructions.  As  part  of  the  survey 
development,  the  researchers  surveyed  10  users  who  were  familiar  with  DL. 
The  10  users  were  chosen  with  the  help  of  our  sponsor,  Major  Maureen  Casey 
of  the  Corporate  Information  office.  The  users  were  reported  to  be  familiar 
with  DL  and  had  been  using  the  system  since  its  initial  design  and  testing 
phases.  In  this  respect,  the  users  should  have  been  as  familiar  with  the 
system  at  AFMC  as  were  any  other  users.  The  purpose  of  the  pre-test,  as 
mentioned  in  the  previous  section  was  to  assess  the  content  validity  of  the 
questionnaire. 

The  pre-test  participants  reported  that: 

-  Our  original  instructions  were  rather  short  and  lacked  detail;  assuming 
that  most  all  users  already  knew  how  to  pick  from  a  file  list  box  or  from  a 
pull-down  menu. 
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-  The  survey  actually  looked  quite  different  depending  on  the  particular 
chosen  monitor's  resolution  and  as  a  result  there  was  confusion  as  to  the 
values  of  the  scales. 

-  We  might  do  better  to  completely  drop  all  reference  to  our  sponsoring 
agency  and  the  fact  that  this  was  an  AFIT  thesis  effort. 

-  The  newly  'simplified'  instructions  were  now  quite  LENGTHY  and  might 
require  printing  out  to  follow  clearly  and  avoid  keyboard  gymnastics. 

Based  upon  this  input: 

-  the  instructions  were  "beefed  up"  to  include  all  menu  choices  and 
intermediate  message  boxes  that  a  user  might  be  confronted  with,  along 
with  the  appropriate  choice  of  responses. 

-  the  instructions  were  modified  to  include  a  clarification  of  the  intent  of  the 
rating  scales  in  case  it  was  difficult  to  figure  out  visually. 

-  the  division  chiefs  name  was  quoted  as  authority  for  the  survey  and  no 
other  explanation  was  given. 

-  instructions  on  printing  the  instruction  sheet  and  a  suggestion  to  do  so  were 
added. 

Pre-Test  Results--Survev  Questions.  Other  than  the  previous 
discrepancies,  which  dealt  wholly  with  the  survey  instructions,  the  pre-test 
participants  reported  that  questionnaire  was  clear,  understandable,  and 
appeared  to  measure  user  satisfaction  with  DL  as  intended.  These  results 
demonstrated  the  content  validity  of  the  research  instrument. 

Reliability  Analysis.  The  questionnaire  was  tested  using  Cronbach's 
alpha  coefficient  to  test  the  variables  of  interest  that  made  up  our  measure  of 
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user  satisfaction.  The  variables  are  listed  along  with  the  alpha  values  in  the 
following  table. 


Table  4 

RELIABILITY  DATA  FOR  VARIABLES  OF  INTEREST 


Variable 

Questions 

Alpha 

Overall  Satisfaction 

11,  13,  22 

.89 

14,  17,  19-21,  28,  29 

.91 

12,  18 

.33 

Training/Learning 

15,  16,  25,  27* 

.45* 

User  Interface 

23,  24,  26 

.85 

*  The  alpha  value  for  Training  was  improved  to  .60  by  removing  question  27 
from  consideration. 

Survey  Administration.  Based  upon  the  results  of  the  pre-test,  the 
survey  instrument  was  finalized  and  administered  to  75  users  in  SI  and  10 
users  in  Cl  at  AFMC.  Of  those  surveyed,  8  SI  users  and  4  Cl  users  returned 
usable  surveys  for  a  response  rate  of  16  percent  of  those  surveyed. 

Population.  The  test  population  for  the  customer  satisfaction  survey 
consists  of  seventy-five  members  of  the  AFMC  staff  who  are  responsible  for 
developing  software  systems  for  use  in  the  command  and  throughout  the  Air 
Force.  This  population  is  not  merely  a  sample  since  it  constitutes  the  entire 
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population  of  DL  users  as  of  the  date  of  the  survey  administration.  Software 
developers  do  not  plan  to  distribute  DL  Air  Force  wide  until  they  are 
satisfied  that  the  product  is  capable  of  performing  up  to  expectations. 

Survey  Results--Demographics.  The  respondents  were  mostly  civilian 
male  employees,  in  their  thirties  or  forties,  and  having  some  college 
background.  Those  with  degrees  mostly  had  Bachelor's  degrees.  All 
participants  had  over  four  years  using  computers,  and  most  had  more  than 
ten.  For  most,  this  was  the  first  electronic  records  management  tool  that 
they  had  used,  and  they  received  on-the-job  training  rather  than  formal 
classroom  training.  Half  of  the  participants  had  used  DL  for  less  than  30 
days.  Another  quarter  had  it  for  more  than  90  days,  and  the  rest  were 
somewhere  in  between.  This  demographic  data  is  summarized  in  Appendix 
G. 

Survey  Results 

This  section  describes  the  data  collected  by  the  survey  instrument  in 
accordance  with  the  methodology  provided  in  the  previous  section.  These 
results  are  given  in  the  form  of  descriptive  statistics. 

Survey  Results-Statistics  bv  Variable  of  Interest.  On  the  average  the 
data  supported  no  significant  statement  about  user  satisfaction.  As  Table  5 
shows,  all  means  were  between  3.0  and  5.0  with  standard  deviations  of  less 
than  2.0. 
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Table  5 

DESCRIPTIVE  STATISTICS  BY  VARIABLE  OF  INTEREST 


DESCRIPTIVE 

STATISTICS 

Mean 

Standard 

Deviation 

Scale 

Mean 

Scale 

Standard 

Deviation 

Overall  Satisfaction 

QUEST11 

3.33 

1.37 

3.47 

0.24 

QUEST13 

3.75 

1.48 

QUEST22 

3.33 

1.37 

Training/^ 

QUEST15 

4.83 

1.53 

4.29 

0.60 

QUEST16 

3.50 

1.38 

QUEST25 

4.67 

0.98 

QUEST27 

4.17 

0.72 

Reliability 

QUEST12 

3.58 

1.56 

3.88 

0.42 

QUEST18 

4.17 

1.40 

User  In 

terface 

QUEST23 

4.75 

1.22 

4.55 

0.21 

QUEST24 

4.58 

1.38 

QUEST26 

4.33 

1.72 

Productivity 

QUEST14 

4.00 

1.54 

3.77 

0.40 

QUEST17 

3.33 

1.15 

QUEST19 

4.17 

1.90 

QUEST20 

4.25 

1.76 

QUEST21 

3.92 

1.38 

QUEST28 

3.42 

1.38 

QUEST29 

3.33 

1.37 

Where  the  standard  deviations  were  less  than  1.0,  indicating  better 
agreement,  the  questions  dealt  with  ease  of  learning  and  lack  of  frustration 
with  the  system  as  a  whole  (questions  25  and.27).  A  population  with  at  least 
four  years  and  often  more  than  ten  years  of  computer  experience  could  be 
expected  not  to  have  much  difficulty  with  a  new  system.  Grouping  questions 
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by  their  common  variables  of  interest,  namely  Overall  Satisfaction  level, 
Productivity  improvements,  Reliability  of  the  software,  Training  and 
Learning  ease,  and  User  Interface  also  revealed  no  significant  trends  for 
further  investigation. 


IV.  CONCLUSIONS  AND  RECOMMENDATIONS 

This  chapter  contains  the  conclusions  and  recommendations  of  this 
research.  The  thesis  objectives  were  to  generate  a  definition  of  effectiveness 
for  the  Records  management  (RM)  field  and  to  determine  the  impact  that 
Document  Librarian  (DL)  has  made  on  the  RM  process  for  the  software  test 
population.  Two  specific  investigative  questions  were  identified.  The 
following  sections  provide  an  overview  of  the  findings. 

Conclusions 

Investigative  Question  1  How  is  effectiveness  defined  for  RM? 

The  research  team  employed  a  Delphi  process  as  described  in  Chapters 
II  and  III  to  generate  a  consensus  among  Air  Force  experts  in  the  RM  field  in 
order  to  define  RM  effectiveness.  Four  rounds  were  required  before  consensus 
was  reached. 

The  results  of  the  Delphi  emphasize  five  primary  functions  that  a  RM 
system  should  provide  for  its  users.  The  functions  are  storing,  screening, 
retrieving,  disposition,  and  indexing.  These  five  main  functions  provide  the 
basis  for  defining  effectiveness  for  a  RM  system,  according  to  the  expert 
panel  in  the  Delphi.  In  order  for  a  system  to  effectively  manage  records,  it 
must  have  the  capability  to  perform  all  these  functions  well.  The  function  of 
storing  records  includes  reducing  the  possibility  of  loss  and  facilitating  ease 
of  recovery  of  documents.  Screening  involves  providing  access  to  records  in  a 
logical  and  systematic  way  so  the  records  manager  is  able  to  scan  records  for 
requirements  imposed  by  Privacy  Act  and  Freedom  of  Information  Act 
statutes.  Retrieving  records  allows  searches  of  documents  by  a  predetermined 
field  to  retrieve  a  desired  subset  of  all  records  for  review.  Disposition 
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instructions  are  regulatory  plans  for  archival  storage  and  eventual 
destruction  of  all  records.  These  instructions  are  also  linked  with  a  record's 
indexing  scheme  so  that  records  can  be  grouped  by  disposition  requirements. 

The  indexing  function  requires  the  creation  and  maintenance  of  indexes 
based  on  characteristics  such  as  key  words,  document  subject,  full  text,  or 
disposition  instructions. 

Investigative  Question  2  How  does  Document  Librarian  impact  the 
process  of  RM? 

The  method  chosen  to  evaluate  the  effectiveness  of  electronic 
technology  was,  as  described  in  Chapters  II  and  III,  a  customer  satisfaction 
survey  of  users  of  the  new  RM  software  package  DL.  The  test  population  was 
AFMC/CI  and  SI  personnel  who  have  been  using  and  testing  the  software  for 
the  command. 

The  survey  results  suggest  that  the  population  testing  DL  has 
experienced  no  dramatic  improvement  or  detriment  as  a  result  of  the 
software  addition  to  the  organization's  RM  process.  Respondents  indicated 
that  DL  has  not  improved  their  job  productivity:  in  fact,  they  have  been 
slightly  dissatisfied  with  the  productivity  changes  since  DL  was 
implemented.  They  also  have  indicated  by  their  overall  satisfaction  that  they 
are  unconvinced  of  DL's  usefulness  as  a  tool  in  their  RM  process.  Overall 
survey  responses  indicated  they  were  mildly  dissatisfied  with  DL,  responding 
that  the  software  was  not  particularly  easy  to  use  or  helpful  in  saving  them 
time  and  did  not  significantly  contribute  to  improved  productivity  in  their  > 

RM  tasks. 
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Recommendations 


AFMC/CI  personnel  (the  sponsors  of  both  the  development  of  DL  and 
this  research)  have  maintained  high  hopes  that  DL  would  facilitate  a  great 
improvement  in  the  organization’s  RM  productivity.  This  research  has  been 
unable  to  either  confirm  or  disavow  those  hopes.  Members  of  the  thesis  team 
believe  that  further  research  is  necessary,  however,  after  two  main  steps  are 
taken  to  improve  measurement  of  results. 

First,  DL  should  be  used  for  a  longer  period  of  time  before  any 
evaluation  is  made.  Several  of  the  survey  respondents  had  not  used  DL  for 
more  than  30  days  prior  to  being  questioned.  Initial  dissatisfaction  with  new 
work  tools  like  DL  is  common  on  the  job,  and  a  learning  curve  can  also  be 
expected.  With  these  things  in  mind,  conducting  a  satisfaction  survey  so  soon 
after  DL's  implementation  was  probably  not  practical.  Workers  need  time  to 
adjust  to  the  new  way  of  doing  their  jobs  and  to  become  completely  familiar 
with  the  functions  of  the  software. 

Second,  a  larger  number  of  users  should  be  surveyed  in  order  to  ensure 
a  sample  which  is  more  representative  of  the  population.  Clearly,  the 
response  was  disappointing  and  needs  further  study.  Just  prior  to  the 
administration  of  the  survey,  the  test  population  was  disbanded  and 
redistributed  across  AFMC  to  several  locations.  This  occurrence  was  probably 
a  major  factor  in  limiting  the  number  of  respondents  to  the  survey.  Another 
reason  why  the  number  of  user  surveys  returned  was  much  less  than 
expected  may  in  some  part  be  due  to  the  method  being  used  to  conduct  it.  The 
ease  with  which  an  electronic  survey  can  be  ignored  and  the  ease  with  which 
it  can  be  answered  are  similar,  and  the  lengthy  instructions  provided  (for 
simplicity's  sake)  may  have  intimidated  some.  Once  stability  has  returned  to 
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the  DL  user  population,  and  users  have  had  several  months  experience  with 
the  software,  a  follow-up  survey  should  be  conducted.  However,  any  further 
evaluation  will  require  much  planning  and  interoffice  cooperation  to 
successfully  carry  out  the  research. 

Since  the  Delphi  identified  several  key  functions  necessary  for  a  RM 
system  to  be  effective,  any  subsequent  evaluation  of  DL  should  include  those 
functions  as  a  part  of  the  test.  The  researchers  recommend  that  the 
effectiveness  criteria  labeled  by  the  experts  as  critical  be  added  to  the  survey 
instrument  to  test  whether  or  not  DL  satisfies  those  criteria.  Although  the 
criteria  were  developed  at  the  same  time  that  the  survey  was  being 
conducted,  it  is  possible  that  the  results  of  the  survey  would  have  been  more 
positive  had  these  criteria  been  included  as  part  of  the  questioning.  This 
would  further  evaluate  the  usefulness  of  the  software. 

The  thesis  team  also  recommends  that  APMC/CI  evaluate  other 
software  packages  to  avoid  generalizing  all  RM  software  by  the  evaluation  of 
DL.  If  the  command  remains  interested  in  developing  a  system  to 
electronically  manage  records,  the  system  developers  should  not  limit  the 
decision  makers  to  the  sole  choice  of  DL.  Automation  is  certainly  possible 
with  other  software. 

One  final  recommendation  requires  a  very  different  approach  than  has 
previously  been  taken.  Even  though  preliminary  results  suggest  that  DL  has 
not  improved  overall  productivity  of  RM,  the  problem  may  not  be  with  the 
software.  DL  is  designed  simply  to  automate  an  already  existing,  manual 
system  of  managing  records,  but  what  may  be  needed  is  an  entirely  new 
method  of  doing  the  job.  AFMC/CI  is  advised  to  perform  a  functional  process 
improvement  (FPI)  procedure  to  completely  reevaluate  the  Air  Force  way  of 
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RM.  The  purpose  and  reward  of  a  FPI  is  to  reevaluate  current  processes  and 
to  engineer  new  and  better  methods  of  accomplishing  those  processes.  If  the 
current  RM  process  is  not  as  effective  as  it  could  be,  merely  automating  that 
ineffective  process  cannot  realistically  be  expected  to  improve  productivity. 
The  time  to  develop  automation  software  is  after  AFMC/CI  has  developed  an 
overall  effective  and  productive  RM  process  through  a  FPI. 
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Appendix  A:  Rating  Instructions  and  Description  of  Functions 


Instructions: 

A.  Please  rate  the  importance  of  the  following  functions  within  the 
records  management  process.  Rate  each  function  once,  then  add  any  extra 
functions  you  regard  as  important  to  the  process.  Please  indicate  your  choice 
in  the  space  provided  (in  column  ONE  on  the  email). 

How  important  is  each  of  the  following  to  the  successful 
accomplishment  of  records  management?  Please  answer  with  one  of  these 
choices: 

(unimportant)  (very  important) 

1 . 2 . 3 . 4 . 5 . 6 . 7 

(undetermined) 

_ a.  Filtering  information:  filtering  personal  from  professional  information 

and  converting  from  various  formats  to  an  accepted  standard 

_ b.  Making  records:  generation  of  standard  reports  with  suspense  dates, 

Freedom  of  Information/Privacy  Act  requirements,  efficient  tracking  of 
revisions,  and  integration  of  medium  formats  (optical  disk,  magnetic 
disk,  etc.)  into  a  single  record 

_ c.  Assigning  disposition  instructions  automatically 

_ d.  Indexing  records  automatically 

_ e.  Storing  records:  to  eliminate  the  possibility  of  loss  and  allowing  for 

ease  of  retrieval 

_ f.  Screening  records:  allow  for  easy  access  to  records 

_ g.  Retrieving  records:  allow  retrieval  by  predetermined  fields  and  confine 

access  to  records  as  needed 

_ h".  Allowing  the  transfer  of  records  to  permanent  storage  sites  and 

maintaining  a  detailed  audit  trail 

_ i.  Destroying  records  automatically,  in  accordance  with  prescribed 

retention  directives 
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Instructions: 

B.  In  this  section,  please  give  additional  comments  about  the  records 
management  functions  listed  above  i.e.:  why  each  function  is  important  and 
what  important  auxiliary  information  should  be  considered  for  the  functions. 
Your  supplementary  observations  are  a  vital  part  in  the  acquisition  of  useful 
research  data. 
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Appendix  B:  Ranking  Instructions 


INSTRUCTIONS 
A.  To  rank  order: 

In  the  blank  next  to  the  statement  number,  place  the  number  that 
corresponds  to  your  opinion  of  the  rank  order  of  that  statement  in  relation  to 
the  other  statements  provided.  Use  numbers  1-9  , 

but  use  each  number  only  once.  Number  1  represents  the  most  important; 
number  9  represents  the  least  important.  Also,  at  the  end  of  the  list  of 
statements  we  have  provided  space  for  you  to 

make  any  comments  you  feel  appropriate  for  other  Delphi  participants  to 
consider  in  making  their  decisions. 

_ a.  Filtering  information:  filtering  personal  from  professional  information 

and  converting  from  various  formats  to  an  accepted  standard 
Round  2  -  Average:  5.0  Range:  2  to  7 

_ b.  Making  records:  generation  of  standard  reports  with  suspense  dates, 

Freedom  of  Information/Privacy  Act  requirements,  efficient  tracking  of 
revisions,  and  integration  of  medium 

formats  (optical  disk,  magnetic  disk,  etc.)  into  a  single  record 
Round  2  -  Average:  5.7  Range:  5  to  6 

_ c.  Assigning  disposition  instructions  automatically 

Round  2  -  Average:  5.6  Range:  3  to  7 

_ d.  Indexing  records  automatically 

Round  2  -  Average:  5.9  Range:  4  to  7 

_ e.  Storing  records:  to  eliminate  the  possibility  of  loss  and  allowing  for 

ease  of  retrieval 

Round  2  -  Average:  6.9  Range:  6  to  7 


_ f.  Screening  records:  allow  for  easy  access  to  records 

Round  2  -  Average:  6.3  Range:  6  to  7 


_ g.  Retrieving  records:  allow  retrieval  by  predetermined  fields  and  confine 

access  to  records  as  needed 

Round  2  -  Average:  6.1  Range:  3  to  7 
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_ h.  Allowing  the  transfer  of  records  to  permanent  storage  sites  and 

maintaining  a  detailed  audit  trail 
Round  2  -  Average:  5.9  Range:  5  to  7 


_ i.  Destroying  records  automatically,  in  accordance  with  prescribed 

retention  directives 

Round  2  -  Average:  4.6  Range:  1  to  7 


B.  In  this  section,  please  give  additional  comments  about  your  opinions 
listed  above  (if  desired). 

Comments — 
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Appendix  C:  Results  of  Delphi 
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Appendix  E:  Survey  Questions 


DOCUMENT  LIBRARIAN  SURVEY  (PAGE  1) 


SECTION  I.  BACKGROUND  PLEASE  PUT  THE  CORRECT  NUMBER  FOR  YOUR  RESPONSE  IN  THE  BOX  NEXT  TO 
THE  QUESTION 


1.  WHAT  IS  YOUR  GENDER? 

1.  MALE 

2.  FEMALE 


2.  WHAT  IS  YOUR  AGE? 


□ 


1.  LESS  THAN  25 

2.  26-32 

3.  33-39 

4.  4046  _ 

5.  47-53 

6.  5460 

7.  OVER  60 

WHAT  IS  YOUR  HIGHEST  LEVEL  OF  EDUCATION? 


4.  HOW  LONG  HAVE  YOU  USED  A  COMPUTER? 

1.  LESS  THAN  1  YEAR 

2.  1-3  YEARS 

3.  4 -6 YEARS 

4.  7- 10  YEARS 

5.  MORE  THAN  10  YEARS 

5.  WHAT  IS  YOUR  OFFICE  SYMBOL?  (AFMC  is 
assumed) 


1.  LOWER  THAN  HIGH  SCHOOL 

2.  HIGH  SCHOOL  OR  EQUIVALENT 

3.  SOME  COLLEGE  BUT  NO  DEGREE 

4.  ASSOCIATE  S  DEGREE  OR  CERTIFICATE 

5.  BACHELOR'S  DEGREE 

6.  MASTER'S  DEGREE 

7.  DOCTORAL  DEGREE 
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DOCUMENT  LIBRARIAN  SURVEY  (PAGE  2) 

SECTION  I.  (CONTINUATION)  BACKGROUND  PLEASE  PUT  THE  CORRECT  NUMBER  FOR  YCUR  RESPONSE  IN 
THE  BOX  NEXT  TOTHE  QUESTION. 


6.  IS  DOCUMENT  LIBRARIAN (DL)THE  FIRST  ELECTRONIC 
REOCRDS  MANAGMENT  PROGRAM  TOCLYOU  HAVE 
USED  (EXCEPT  FCRTHE  RECORDS  INFORMATION 
MANAGEMENT  SYSTEM  -  RIMS>7 

1.  YES 

2.  NO 

7.  HAVE  YCU  RECEIVED  FORMAL  (CLASSROOM) 
TRAININGCN  OPERATING  THE  DLSYSTEM? 

1.  YES 

_  _  2.  NO 

|  8.  HAVE  YCU  REC8VEDCN-THE-JC8  (CUT) TRAINING? 


□ 


9.  HOW  LCNGHAVE  YCU  BEEN  USINGDL? 


1.  LESS  T  HAN  30  DAYS 


2.  31-60  CAYS 

3.  61-90  CAYS 

4.  MCRE THAN 90  DAYS 


□ 


10.  WHAT  IS  YCU R  DES IGNAT ICN? 


1.  CIVILIAN 

2.  ENLISTED 

3.  CFFICER 


1.  YES 


DOCUMENT  LIBRARIAN  SURVEY  (PAGE  3) 


SECTION  II.  SATISFACTION  WITH  DOCUMENT  LIBRARIAN  .  PLEASE  PUT  THE  OCRRECT  NUMBER  FCR  YOJR 


RESPONSE  IN  THE  BOX  NEXT  TOTHE  QUESTION. 


□ 


11.  RELEVANCY.  THE  DEGREE  TO  WHICH  a  PRO/0ES  THE  SERVICES  YOU  REOJ1RE 


1 . 2 . 3 . 4 . 5 . 6 . 7 


VERY  NEITHER  VERY 

UNSATISFIED  SATISFIED  NOR  SATISFIED 

DISSATISFIED 

12.  RELIABILITY.  THE  RELIABILITY  OF  DL  (MINIMUM  DOWNTIME  DUE  TO  SOFTWARE  FAILURE) 

1 . 2 . 3 . 4. . 5 . 6. . 7 


VERY  NEITHER  VERY 

UNSATISFIED  SATISFIED  NOR  SATISFIED 

DISSATISFIED 

j  J  13.  COMPLETENESS.  THE  COMPREHENSIVENESS  CF  DL'S  SERVICE  CAPABILITIES. 

1 . 2 . 3. . 4 . 5. . 6 . 7 


VERY  NEITHER  VERY 

UNSATISFIED  SATISFIED  NOR  SATISFIED 


□ 


DISSATISFIED 

14.  VOLUME  CF  OUTPUT.  THE  OUTPUT  CF  INFORMATION  DL  ENABLEB'VCU  TOMANAGE. 


1 . 2 . .....3 . 4 . 5 . 6. . 7 


□ 


VERY  .  NEITHER  VERY 

UNSATISFIED  SATISFIED  NOR  SATISFIED 

DISSATISFIED 

15.  UNDERSTANDING  CF  SYSTEM  HCW  WELL  YOJ  UNDERSTAND  DL  AND  ITS  USE. 

1 . 2 . 3 . 4. . 5 . 6. . 7 


VERY  NEITHER  VERY 

UNSATISFIED  SATISFIED  NOR  SATISFIED 

DISSATISFIED 


> 


W 
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DOCUMENT  LIBRARIAN  SURVEY  (PAGE  4) 

SECTION  II.  (CONT INUATION).  SATISFACTION  WITH  DOCUMENT  LIBRARIAN.  PLEASE  PUT  THE  OCRRECT 
NUMBER  FCR  YCUR  RESPONSE  IN  THE  BOX  NEXT  TOT  HE  QUESTION. 

16.  DEGREE  OF  TRAINING  THE  QUALITY  CF  INSTRUCTION  YOU  RECEIVED  TOCEVELCP 
YCUR  PROFICIENCY  IN  USINGTHE  DL  SYSTEM. 


1 . 2 . 

. 3 . 4 . 

....5 . 

. 6 . 

VERY 

NEITHER 

VERY 

UNSATISFIED 

SATISFIED  NOR 
DISSATISFIED 

SATISFIED 

□ 


17.  JOB  EFFECTS.  THE  CHANGES  IN  JC6  PERFORMANCE  RESULT ING FROM  THE 
IMPLEMENT  AT  104  CF  THE  DL  SYST  EM 


1 . 2 . 

. 3 . 4 . 

....5 . 

. 6 . 

VERY 

NEITHER 

VERY 

UNSATISFIED 

SATISFIED  NOR 

SAT  iSFIED 

DISSATISFIED 

□ 


18.  SECURITY  OF  INFORMATION.  THE  SAFEGUARDING  CF  INFORMATION  FROM  MISAPPROPRIATION 
CR  UNAUTHORIZED  ALTERATION  CR  LCSS. 


1 . 2. 

VERY 

UNSATISFIED 


,3 . 4. . 5 . 6 . 7 

NEITHER  VERY 

SATISFIED  NOR  SATISFIED 

DISSATISFIED 


* 
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DOCUMENT  LIBRARIAN  SURVEY  (PAGE  5) 

SECTION  II.  (CONT INUATION).  SATISFACTION  WITH  DOCUMENT  LIBRARIAN.  PLEASE  PUT  THE  CORRECT 
NUMBER  FCR  YCUR  RESPONSE  IN  THE  BOX  NEXT  TOT  HE  QUESTION. 


1  1  19. 


PERCEIVED  UTILITY.  YCUR  JUDGEMENT  ABOUT  THE  USfTULNEBS  OR  THE  PRODUCTIVITY  OFT  HE  DL  SYSTEM. 


1 . 2 . 

. 3. . 4 . 

....5 . 

. 6. . 7 

VERY 

NEITHER 

VERY 

UNSATISFIED 

SAT  ISFIED  NOR 
DISSATISFIED 

SATISFIED 

□  so. 


CONVENIENCE  OF  ACCESS.  THE  EASE  CF  ACCESS  CAPABILITY  WITH  DL. 


1 . 2 . . 

. 3 . 4 . 

...5 . 

. 6 . 

VERY 

NEITHER 

VERY 

UNSATISFIED 

SATISFIED  NOR 

SATISFIED 

DISSATISFIED 

□ 


21.  INTEGRATION  OF  SYSTEMS.  T  HE  ABILITY  CF  DL  TOT  RANSMIT  INFORMATION  BETWEEN 
SYSTEMS  SERVICING  FUNCTIONAL  AREAS  CR  OTHER  LOCATIONS. 


1 . 2 . 

. 3 . 4 . 

....5. . 

. 6 . 

VERY 

NEITHER 

VERY 

UNSATISFIED 

SAT  ISFIED  NOR 

SAT  ISFIED 

DISSAT  ISFIED 
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DOCUMENT  LIBRARIAN  SURVEY  (PAGE  6) 

SECTION  III.  OVERALL  APPRAISAL.  PLEASE  PUT  T HE  CORRECT  NUMBER  FOR  YOUR  RESPONSE  IN 
THE  BOX  NEXT  TO  THE  QUESTION. 

|  |  22.  OVERALL  SAT  ISF ACTION  WIT  H  DL. 

1 . 2 . 3 . 4 . 5 . 6 . 7 


VERY 

UNSATISFIED 


NEITHER 
SATISFIED  NOR 
DISSATISFIED 


,3 . 4. . 5. 


VERY 

SATISFIED 


.6 . 7 


DIFFICULT 


EASY 


□ 


24.  THE  USER  INTERFACE  CF  DL  IS: 


1 . 2 . 3 . 4 . 5 . 6. . 7 

CONFUSING  CLEAR 


1 


* 
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DOCUMENT  LIBRARIAN  SURVEY  (PAGE  7) 


SECTION  III.  (CONTINUATION) OVERALL  APPRAISAL.  PLEASE  PUT  THE  CORRECT  NUMBER  FCR 
YCUR  RESPONSE  IN  THE  BOX  NEXT  TOTHE  QUESTION. 


1 _ 125. 

1 . 2 . 

HARD  TO 

LEARN 

. 3 . 

. 4 . 

. 6 . 7 

EASY  T  O 

LEARN 

1 _ ]  26. 

1 . 2 . 

. 3 . 

. 4 . 

. 5 . 

. 6 . 7 

□  27 

IMPERSONAL 

1 . 2 . 

. 3 . 

. 4 . 

. 5 . 

FRIENDLY 

. 6 . 7 

□  ». 

FRUSTRATING 

1 . 2 . 

. 3 . 

. 4. . 

. 5 . 

NOT  FRUSTRATING 

. 6 . 7 

1  I29. 

TIME  WASTING 

1 . 2 . 

. 3 . 

. 4. . 

. 5 . 

TIME  SAVING 

. 6 . 7 

UNPRODUCTIVE 

PRODUCTIVE 

y 


* 
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Appendix  F:  Survey  Instructions 


This  is  a  research  instrument  commissioned  by  AFMC/CIMR  to  evaluate  the 
effectiveness  of  Document  Librarian. 

This  is  an  OFFICIALLY  SANCTIONED  Survey  under  the  authority  of  Mr. 
Mike  Riley,  MSC/SI. 

Please  respond  within  5  working  days  from  your  receipt  of  this  message. 

Thank  You  for  taking  the  time  to  fill  out  our  User  Satisfaction  Survey! 

Surveys  are  an  IMPORTANT  research  tool,  and  we  are  attempting  to  make  it 
as  convenient  as  possible  for  the  user  AND  researcher  by  doing  it 
electronically! 

*NOTE*  You  may  wish  to  print  these  instructions  for  convenience  in 
following 

them;  either  click  on  the  Print  button  or  choose  FILE  from  the  menu  bar  and 
PRINT  from  the  pull-down  menu. 

1.  Attached  to  this  message  you  will  find  the  survey  form  "DOCLIB.FRP". 

2.  View  the  attachments  by  a  single  click  on  the  PAPER  CLIP  symbol. 

3.  At  the  new  window,  make  sure  your  U:  drive  is  selected  in  the  bottom 
right  comer  under  Drives.  If  it  is  not,  click  on  the  arrow  beside  the 
Drive  box  and  then  select  U:. 

4.  Now  click  on  EXTRACT,  which  will  extract  a  copy  of  the  file  to  your  U: 
drive. 

5.  You  will  receive  a  message  that  the  file  was  extracted. 

6.  Click  on  OK. 

7.  Click  on  Done. 

8.  Exit  Beyond  Mail  and  return  to  Windows  Program  Manager. 

9.  Select  and  run  the  PerForm  Pro  Filler  program. 

10.  Select  FILE  from  the  PerForm  Pro  menu  bar. 

11.  Select  OPEN  FORM  from  the  pull-down  menu. 

12.  Make  sure  the  Path  is  ”U:V  and  the  file  is  "DOCLIB.FRP".  If  not, 
either  select  or  type  them  in. 

13.  Click  on  OK. 

14.  Select  DATA  from  the  PerForm  Pro  menu  bar. 

15.  Select  OPEN  DATA  FILE  from  the  pull-down  menu. 

16.  In  the  'Open  Data  File"  dialog  box,  make  sure  the  Path  is  "U:\*.DBF"  and 
the  file  is  "DOCLIB.DBF".  If  not,  type  them  in.  Make  sure  the  format  is 
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set  to  dBASE.  If  the  format  is  set  to  ASCII,  change  the  format  to  dBASE  by 
selecting  it. 

17.  Click  on  OK. 

18.  In  response  to  the  message  stating  "Data  file  not  found.."  click  YES. 

**  If  you  don't  get  this  message  you  somehow  already  have  the  data  file, 
which  can  lead  to  PROBLEMS!  Exit  PerForm  Filler  now  and  run  FILE 
MANAGER,  locate  the  file  U:\DOCLIB.DBF  and  delete  it  (press  the 
DELETE  key  and  answer  yes  to  the  verification,  if  any),  then  return  to 
PerForm  Filler  and  resume  with  step  #9.  ** 

19.  In  response  to  the  new  message  stating  "New  Database  created"  click  OK. 

20.  **  you  should  see  "DOCLIB.FRP(DOCLIB.DBF)"  on  the  Title  Bar  at  the 
top  of  your  screen;  this  is  the  ONLY  way  your  responses  will  be  saved!  ** 

21.  You  will  now  be  presented  with  the  survey  form  on  the  screen. 

Depending 

on  your  screen,  you  may  not  be  able  to  see  the  first  page  completely. 

22.  Select  VIEW  from  the  top  menu  and  FIT  SIDES  from  the  pull-down  menu 
to  adjust  the  form  to  fit  your  screen.  You  may  wish  to  enlarge  the  view  for 
reading  ease  by  selecting  VIEW  REAL  SIZE  or  VIEW  ENLARGE. 

23.  The  black  boxes  are  for  your  answers  and  the  questions  are  adjacent  to 
each  box. 

24.  Place  the  appropriate  answer  in  the  first  box  and  then  either  carriage 
return  or  arrow  down  or  TAB  to  the  next  one. 

25.  If  you  make  a  mistake  or  change  your  mind  you  can  arrow  back  to  the 
question,  hit  BACKSPACE  and  then  re-enter  the  answer. 

26.  **If  the  scales  in  sections  two  and  three  do  not  exactly  line  up  with 
the  responses  (different  screens  once  again  produce  different  results),  it 
is  intended  that  the  far  left  side  of  the  scale  represents  a  "1",  the 
midpoint  represents  a  "4",  and  the  far  right  side  a  "7"** 

27.  When  you  get  to  question  29  (there  are  a  total  of  29  questions  in  3 
sections  on  7  pages)  you  are  finished!! 

28.  You  can  wrap  from  the  last  question  back  to  the  first  (with  an 
additional  carriage  return/down  arrow/TAB)  to  review  your  answers,  or  use 
PageUP  and  PageDOWN  to  look  at  specific  pages. 

29.  To  save  your  answers,  from  the  top  menu  select  DATA  and  from  the 
pull-down  menu  select  ADD  RECORD. 

30.  Exit  the  PerFORM  Pro  Filler  program  and  return  to  Beyond  Mail. 

31.  Initiate  a  reply  to  the  initial  survey  message  by  highlighting  the 
message  and  clicking  on  the  REPLY  icon. 

32.  Create  an  attachment  by  a  single  click  on  the  PAPER  CLIP  symbol. 

33.  At  the  new  window,  make  sure  your  U:  drive  is  selected  in  the  bottom 
right  comer  under  Drives.  If  it  is  not,  click  on  the  arrow  beside  the 
Drive  box  and  then  select  U:. 


34.  Select  the  file  "DOCLIB.DBF"  and  click  on  ATTACH. 


»• 
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35.  Click  on  OK. 

36.  Send  the  message  on  its  way  by  clicking  on  the  SEND  icon. 

ANY  questions?  or  if  you  don't  know  where  to  send  replies,  address  them  to 
'MOSELEY@wpgatel.wpafb.af.mir. 

(Peter  M.  Moseley) 


Appendix  Q;  Survey  Data 


♦ 


In  the  tables  that  follow,  background  data  is  presented  with  the 
possible  responses  listed  in  the  first  column  and  the  frequency  of  those 
choices  listed  in  the  second  column.  Microsoft  Excel's  histogram  function  was 
used  to  group  the  data  by  response  and  gather  the  frequencies. 

Section  I 


Table  1 


GENDER 

Male 

9 

Female 

3 

Table  2 


AGE 

Less  than  25 

1 

26-32 

1 

33  -  39 

4 

40-46 

4 

47-53 

1 

54-60 

1 

Over  60 

0 

Table  3 


EDUCATION 

Lower  than  High  School 

0 

High  School  or 
Equivalent 

0 

Some  College  but  no 
degree 

4 

Associate's  Degree 

1 

Bachelor's  Degree 

5 

Master's  Degree 

2 

Doctoral  Degree 

0 

60 


Table  4 


* 


> 


COMPUTER  USE 

Less  than  1  year 

0 

1  -  3  years 

0 

4  -  6  years 

2 

7-10  years 

2 

More  than  10  years 

8 

Table  5 

EXPERIENCE  & 
TRAINING 

First 

Electronic 
RM  tool  ? 

HfH 

On-the-job 
Training  ? 

YES 

9 

2 

10 

NO 

3 

10 

2 

Table  6 

USE  OF  DL 

Less  than  30  days 

6 

31-60  days 

2 

61-90  days 

1 

More  than  90  days 

3 

Table  7 


DESIGNATOR 

Civilian 

10 

Enlisted 

2 

Officer 

0 

For  consistency  with  the  data  in  Section  I,  we  have  included  two  tables 
listing  the  question  number  in  the  first  column  and  the  frequencies  of  each 
possible  response  in  the  remaining  columns.  The  data  are  separated  into 


61 


tables  representing  Section  II  (Satisfaction  with  Document  Librarian)  and 
Section  III  (Overall  Appraisal)  in  the  survey. 


* 


4 

Section  II 


Tal 

ale  8 

RESPONSE 

1 

2 

3 

4 

5 

6 

7 

QUEST11 

2 

1 

2 

5 

2 

0 

0 

QUEST12 

1 

2 

3 

3 

1 

2 

0 

QUEST  13 

1 

2 

1 

4 

3 

1 

0 

QUEST14 

1 

1 

1 

5 

3 

0 

1 

QUEST15 

0 

1 

1 

3 

3 

2 

2 

QUEST16 

1 

2 

2 

5 

1 

1 

0 

QUEST17 

1 

2 

2 

6 

1 

0 

0 

QUEST18 

1 

0 

2 

4 

3 

2 

0 

QUEST19 

2 

1 

0 

3 

2 

4 

0 

QUEST20 

2 

0 

i 

2 

4 

3 

0 

QUEST21 

1 

1 

1 

5 

3 

1 

0 

Section  III 


Table  9 


RESPONSE 

1 

2 

3 

4 

5 

6 

7 

QUEST22 

2 

1 

2 

5 

2 

0 

0 

QUEST23 

0 

0 

2 

3 

4 

2 

1 

QUEST24 

1 

0 

0 

4 

4 

3 

0 

QUEST25 

0 

0 

1 

5 

3 

3 

0 

QUEST26 

2 

0 

0 

3 

4 

3 

0 

QUEST27 

0 

0 

2 

6 

4 

0 

0 

QUEST28 

2 

1 

1 

6 

2 

0 

0 

QUEST29 

2 

1 

2 

5 

2 

0 

0 

The  range  of  responses  given  in  the  following  table  shows  little  more 
information  than  the  descriptive  statistics.  Where  there  is  better  agreement, 
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there  is  an  equally  small  range  of  responses,  showing  that  the  high  degree  of 
computer  literacy  and  the  small,  reasonably  well  educated  population  had 
little  difficulty  adjusting  to  the  system.  Based  on  range  alone,  question  23 
dealing  with  ease  of  use  was  the  most  favorable  (no  response  less  than  a 
value  of  three). 


RANGE  OF 
RESPONSES 


UEST11 


UEST12 


UEST13 


UEST14 


UEST15 


UEST16 


UEST17 


UEST18 


UEST19 


UEST20 


UEST21 


UEST22 


UEST23 


UEST24 


UEST25 


UEST26 


UEST27 


UEST28 


UEST29 


Table  11 


Minimum  Maximum 


Y 


> 
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